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Executive Summary 


This research project is the fifth in a series that has focused on exploring the concept of “generic 
sectors.” A generic sector is defined as one that can be managed by air traffic controllers without the 
need for extensive memorization of sector data or specialized skills. This Next Generation Air 
Transportation System (NextGen) concept could support a more flexible allocation of the controller 
workforce. Current Federal Aviation Administration (FAA) NextGen plans include generic sectors as 
part of the FAA’s High Altitude Airspace Concept. 

Previous generic airspace research has made use of a Controller Information Tool (CIT), which 
displays data a controller may need to work traffic in a specific sector. Also included were automation 
tools such as DataComm and conflict detection/resolution, which also support the generic concept. 

The current experiment was conducted at the NASA Ames Research Center and was designed to 
investigate the placement of the CIT display. The alternatives included “Standalone,” with the CIT on 
a separate screen above the control position, and “Integrated,” with the information directly accessible 
on the radar display. 

The simulation laboratory research environment was Oakland Center sectors 30 and 33 combined. 
Following training on the CIT and automation tools, sixteen recently retired FAA controllers each 
worked two traffic scenarios using the two versions of the CIT. Each scenario ran for 50 minutes. The 
traffic consisted of 40% to 50% DataComm-equipped aircraft. Data were recorded on workload, 
frequency of access to CIT data, operational errors, CIT placement preference, and CIT usability. 

Workload was assessed using a range of methods including observer ratings, on-screen entries, and 
post-run questionnaires. There was no significant difference between the Standalone and Integrated 
CIT conditions on any of the workload measures. However, the participants expressed a clear 
preference for the Integrated CIT. It reportedly allowed better situation awareness because the 
controllers could keep their focus on the traffic display. 

Logs of CIT key usage showed that sector frequencies were accessed most often. Sector information 
was also called up, but less often. Route keys (e.g., San Francisco traffic flows) were used 
occasionally, and mostly at the beginning of a run. Controllers accessed CIT data about the same 
number of times in each condition. On average, the CIT information was removed from the screen 
(turned off) more often in the Integrated condition, perhaps to control display clutter. Suggestions for 
improvements to the CIT user interface included making the fonts larger, combining the frequency and 
sector keys, and removing the time out feature of the CIT data. 

There were a few operational errors during the runs, but no differences (in number of errors) between 
the two CIT versions. However, controller scan did seem to play a role in some of the Standalone 
errors. Some losses of separation were attributed to difficulties using the automation tools. 

The conclusions of this study are that controllers strongly preferred the Integrated version of the CIT 
over a separate, Standalone screen. The primary reason for this seemed to be that it allowed them to 
remain focused on the traffic situation, whereas the Standalone CIT required them to move their 
attention away from the radar presentation for short periods. Further research should identify the 
information needed by controllers and consider how to integrate the CIT into operational systems. 
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1. Introduction 


In the current air traffic management system, sectors of airspace have unique designs, labeling, and 
associated procedures. These characteristics have evolved over the years to accommodate navigation, 
traffic flow, and safety requirements. However, the result is that controllers may need extensive 
training on sectors they manage, which can make the transfer of staff between areas or facilities 
difficult. As part of the Next Generation Air Traffic Control (NextGen) High Altitude Airspace 
Concept, 1 the Federal Aviation Administration (FAA) has been exploring whether some areas of high 
altitude, en route airspace can be designed to be “generic,” such that controllers could work more 
sectors without significant additional training. 

There are two approaches that have the potential to reduce the training time required for controllers to 
manage airspace. The first is to redesign the airspace such that its shape, route structures, labels, 
procedures, etc. do not require extensive memorization and specialized skills. This approach may 
make at least some proportion of sectors easier to work. 

A second approach is to create automation and decision support tools that provide sector data and help 
to manage traffic. In line with this strategy, a Controller Information Tool (CIT) has been developed 
that displays necessary sector and traffic flow details. This should reduce the time required to leam 
and adapt to a sector by providing information, so that less has to be memorized. Preliminary tests of 
the CIT in prior research studies demonstrated that it was valuable in assisting controllers to work 
sectors of unfamiliar airspace. 

Along these same lines, NASA has also been experimenting with automation tools such as DataComm 
and conflict detection/resolution (CD&R). These applications reduce the amount of data a controller 
has to remember and skills that must be acquired by automating such procedures as handoffs and 
transmission of route changes. They also reduce workload by scanning for conflicts and assisting with 
resolutions solutions. 

In the following sections, past CIT research efforts are briefly described, followed by current study 
objectives. Each previous simulation consisted of a “phase” that examined the CIT and automation 
tools in support of generic airspace. 

1.1. Past Research 

The intent of the generic airspace task is to identify sector design features, information, and tools that 
will enable controllers to leam and manage their airspace. These could be used to help controllers 
move between sectors, adapt to boundary changes, or respond to alterations in traffic flow or airspace 
requirements, as determined by algorithms or procedures driven by traffic volume or sector complexity 
measures. 

To begin understanding the functionalities of automation tools required for this task, four 
complementary generic airspace studies, called Phases 1 through 4, were conducted to explore 
different facets of CIT use. For Phase 1, we developed an evaluation in the context of present-day 
automation and procedures, using the NASA Multi Aircraft Control System (MACS) air traffic control 
simulator and a CIT for a single sector. For Phase 2, we modified the information on the CIT based 
on user input from Phase 1 , added a second sector, and applied mid-term NextGen automation tools to 
create a future air traffic management environment. 

In Phase 3, we evaluated the generic airspace concept by rotating controllers through five, contiguous, 
real-world sectors, four of which they had no prior familiarity and where they relied on the CIT for 
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sector information. For Phase 4, we evaluated controllers, unfamiliar with a single generic sector, 
managing traffic under four different traffic scenarios that included off-nominal conditions. 4 The 
overall results from these studies showed that controllers were able to work traffic in unfamiliar sectors 
with acceptable safety and workload with the aid of the CIT, DataComm, and CD&R, supporting the 
generic airspace concept. 

2. Current Study Objective 

For the Phase 5 study, two versions of the CIT were developed to evaluate a Standalone (separate or 
off-screen) versus an Integrated (on-screen) presentation. The purpose of the study was to examine the 
appropriate placement and functional requirements for the information tool. 

The Standalone CIT display was located above the controller’s radar screen while the Integrated CIT 
incorporated the information into the controller’s radar display. The Integrated CIT had the advantage 
of allowing the controllers to access the sector data without taking their eyes off of the radarscope. 
However, too much data on the radar display could result in visual clutter. In contrast, a separate 
screen could allow a significant amount of information to be shown without the clutter issue, but the 
controllers might find the display distracting when they became busy managing traffic. 

Both CIT versions were designed as methods for reducing the memorization and training required to 
manage air traffic in mid-term, NextGen airspace. All problems simulated a mixed communications 
environment of DataComm-equipped and voice-only aircraft. Details of the experimental design are 
described in the next section. 

3. Method 

3.1. Software and Equipment 

For the study, two versions of the CIT (Standalone and Integrated), designed with identical keys, 
displayed information for the combined sectors 30 and 33 in Oakland Center (ZOA). The Standalone 
CIT was mounted above the main radar screen. (See Figure 1.) 

Each CIT had on-screen buttons (or keys) that accessed different CIT views. (See Figures 2 and 3.) 
The same trackball was used to control both the main radar display and the CIT. The available CIT 
views included: 

1 . Special use airspace (SUA). 

2. Radio frequencies. 

3. Sector names and altitudes. 

4. Ground aids: Locations and names of fixes and NAVAIDs. 

5. Major traffic flows for Departures (DEP), Las Vegas (LAS), Oakland (OAK), Seattle (SEA), 
San Lrancisco (SLO), and San Jose (SJC), showing routes, traffic direction, and exit altitudes. 

The keys allowed participants to call up these data as needed for controlling traffic. The displayed 
information only persisted for 15 seconds before disappearing. The information was automatically 
“toggled off’ for experimental purposes, in order to record the number of times that the participants 
needed specific CIT data, and “toggled” it back on. Participants could manually turn the CIT off prior 
to 15 seconds. 
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Figure 1. Standalone CIT. 

The only exception was the SUAs, which did not automatically disappear. The SUA key was designed 
to be persistent to coincide with how the SUAs are depicted on the controllers’ displays in today’s 
operations. 




Figure 2. Standalone CIT keys. 



Figure 3. Integrated CIT keys. 


Figures 4 and 5 illustrate the Standalone and Integrated CITs. Each CIT was programmed with 
identical information. Sector frequencies and route transition altitudes were adjusted for each run to 
minimize the effect of the participant’s ability to learn the information on the different CIT versions. 

The CIT assessment employed the air traffic control simulators resident in the Crew- Vehicle Systems 
Research Facility Air Traffic Control Laboratory. Located in Building N257 at the NASA Ames 
Research Center, this laboratory has ten air traffic control positions. Using FAA en route consoles 
with 20” X 20” displays, the laboratory also has pseudo-pilot workstations and a voice 
communications system. It uses MACS to realistically simulate the FAA en route air traffic control 
Display System Replacement user interface. (See Figure 6 for the laboratory layout.) 
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Figure 4. Standalone CIT display. 



Figure 5. Integrated CIT display. 


MACS was configured to provide controller-pilot DataComm for clearances (with voice backup) and 
CD&R (with manual resolution capability) for DataComm-equipped aircraft. For non-DataComm 
equipped aircraft, controllers were able to use trial planning with manual CD&R and had an interface 
for entering data into MACS to update flight information, but communicated clearances using voice. 
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Figure 6. Air traffic control laboratory layout. 


3.2. Airspace and Traffic Scenarios 

The simulated airspace consisted of a single sector, which was the result of combining sectors 30 and 
33 in ZOA. Combined sector 30/33 is an en route sector on the eastern side of ZOA at FL240 and 
above. The major traffic flows consisted of arrival streams into SFO, OAK, and SJC airports and, to a 
lesser degree, departure streams from these airports, as well as flights to/from other destinations such 
as SEA or LAS. (See Figure 7.) 



Figure 7. Major traffic flows in ZOA 30/33 combined. 


5 


Two traffic scenarios were created for this experiment. Each was based on actual ZOA traffic. The 
scenarios had comparable traffic patterns and traffic loads that approximated the current Monitor Alert 
Parameter (MAP) value of 20, which applied to ZOA 30/33 combined. 5 Each scenario ran for 50 
minutes. The traffic consisted of 40% to 50% DataComm-equipped aircraft. Routes were designed 
with the assumption that all flights were Automatic Dependent Surveillance Broadcast-equipped and 
made use of published waypoints as opposed to employing the ground-based, jet route structure. One 
additional scenario was created for training. 

3.3. Participants 

Sixteen recently retired (within three years) air traffic controllers were recruited as participants (four 
for each day of the simulation). The participants had the necessary skills to manage the sectors, 
although they had not worked the specific ZOA sector in the past. Four retired supervisory air traffic 
control specialists acted as observers. Eight pseudo-pilots assisted with managing the controls of 
aircraft within the simulation. All participants were paid for their time. 

3.4. Experimental Design 

The study was a within-subjects design with two CIT display conditions: Standalone and Integrated. It 
was conducted during a four-day period. On each day, four controller participants managed traffic in 
the ZOA sector. They worked in “parallel simulation worlds” in which they managed the same sector 
but in different experimental conditions and different traffic scenarios. Each participant conducted two 
data collection runs, one for each CIT condition, resulting in a total of 32 scenarios. The order of the 
runs was counter-balanced across the different conditions. (See Table 1.) 

At the conclusion of the first day’s runs, feedback from participants indicated that they needed more 
training on NextGen tools (DataComm and CD&R) to manage traffic in the scenarios. Based on this 
information, it seemed that the workload might be adequately managed with additional hands-on 
training without making changes to the traffic scenarios. Since modifying traffic scenarios at the last 
minute was undesirable, we added one hour of training for the remaining three days and excluded the 
day 1 data from our analyses. The data from the remaining twelve participants were included in the 
results. 


Table 1 . Experimental matrix for the Phase 5 study (repeated each day for four days). 



Participant 1 

Participant 2 

Participant 3 

Participant 4 

Run 1 

Integrated CIT 

Standalone CIT 

Integrated CIT 

Standalone CIT 

Scenario A 

Scenario B 

Scenario B 

Scenario A 

Run 2 

Standalone CIT 

Integrated CIT 

Standalone CIT 

Integrated CIT 

Scenario B 

Scenario A 

Scenario A 

Scenario B 


3.5. Procedures 

The study was scheduled from March 8 to March 1 1 , 201 1 . On each day, four controller participants 
attended a briefing on the experiment and on how to operate the automation tools in MACS 
(DataComm and CD&R). Each participant was assigned to a separate workstation and provided with a 
hands-on introduction to the CIT. On the second through fourth days, they were then presented with a 
50 minute “tools training” traffic scenario. Controllers used the CIT (both Integrated and Standalone 
were available) to access sector data needed for managing traffic. This was followed by two runs to 
collect performance data. 
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Using a screen playback tool, controllers were then interviewed about how they dealt with each use of 
the CIT user interfaces. NASA staff also asked questions about the usability of the CITs. There was a 
final discussion at the end of the day. The daily schedule is shown in Table 2. 


Table 2. Experiment schedule. 


Time 

March 8 

March 9 

March 10 

March 11 

0800 - 0900 

Briefing 

Briefing 

Briefing 

Briefing 

0900 - 1000 

Tools Training 

Tools Training 

Tools Training 

1000- 1030 

Break 

Break 

Break 

Break 

1030- 1130 

Run 1 

Run 1 

Run 1 

Run 1 

1130- 1200 

Debrief 

Debrief 

Debrief 

Debrief 

1200- 1300 

Lunch 

Lunch 

Lunch 

Lunch 

1330- 1430 

Run 2 

Run 2 

Run 2 

Run 2 

1430- 1600 

Analysis 

Analysis 

Analysis 

Analysis 

1600- 1630 

Discussion 

Discussion 

Discussion 

Discussion 


3.6. Data Collection 

Key metrics for the study were chosen to assess the CITs in order to evaluate their placement. 
Subjective measures included ease-of-use of each display type. The participants also provided 
feedback on whether they preferred the Standalone or Integrated CIT while working traffic. Objective 
CIT usage data were collected and compared between the display types to corroborate the results from 
the subjective preferences. Objective metrics, such as operational errors and controller workload, were 
also gathered. 

The following is a list of dependent measures used for the CIT study: 

1. On-screen Workload Assessment Keypad (WAK) input every five minutes. 

2. Frequency of access to CIT data. 

3. Camtasia screen recordings of the controller’s MACS display to assess CIT usage and safety 
critical events. 

4. Post-run questionnaires (workload and other measures). 

5. MACS computer-recorded metrics (operational errors). 

6. Post-experiment questionnaires. 

7. Over-the-shoulder data collected by controller observers (including workload ratings and 
frequencies used for handoffs). 

8. Post-experiment debriefing to collect additional information. 

4. Results 

The workload results are described first to establish the traffic situation that the controllers 
experienced. This may have had an impact on CIT use during the study. Due to the minimal training 
time allotted prior to data collection and the moderate to heavy traffic levels that were presented to the 
controller participants, they spent a large portion of the simulation runs at or near their maximum 
workload thresholds. After the workload results, the participants’ preferences for the CIT location and 
usage are described in detail, followed by an account of various operational issues that resulted from 
the study. 
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Statistical tests were carried out on the data at the p < .05 level. Only the statistically significant test 
results are reported. Error bars on the graphs represent the standard error of the sample. Fisher’s Least 
Significant Difference test was used for post hoc analyses. 

4.1. Traffic Load and Controller Workload 

Traffic level and complexity for this experiment were based on the MAP value for ZOA sectors 30 and 
33 combined. This is the level of traffic at which ZOA Traffic Management receives an alert that 
traffic is approaching capacity for the sector. 

The MAP, which has a value of 20 for the sector tested, is defined as the number of aircraft predicted 
to be in the sector at the same time. Both traffic scenarios were designed to provide traffic at or near 
the MAP value for the sector. Figure 8 illustrates the aircraft count for the two scenarios used in this 
study. 



Figure 8. Number of aircraft in traffic scenarios A and B during the simulation runs. 

The problems reached 75% of the MAP value six minutes into the 50-minute scenario. The traffic in 
the scenarios reached 90% of MAP 13 minutes after the start of the problem and continued to increase 
to 100% of the MAP or more about 17 minutes after starting the run. Traffic levels remained at 
moderate or greater levels for 88% of the scenario time. Traffic volume, control actions, and the lack 
of familiarity with ZOA 30/33 caused controllers to be more reactive than proactive in their approach 
to sector operations. 

Controller workload was assessed using four methods: WAK ratings, participant post-run workload 
ratings of peak and average workload per simulation run, and over-the-shoulder assessments of 
controller workload by expert observers. (See Figure 9.) The ratings were on a scale of one (low 
workload) to five (high workload). WAK ratings were taken every five minutes during each 
simulation run. For all four methods of workload assessment, the workload values were similar 
between the Standalone and Integrated CIT conditions. Any differences were not statistically 
significant. (The WAK ratings from one participant were excluded from the analyses because s/he 
pressed the “1” key throughout the runs instead of values that reflected his/her workload.) 
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Figure 9. Workload data from the observers, the WAK, and post-run questionnaires. 

4.2. Subjective Preference 

Participants were asked to rate their preferences for the Standalone versus Integrated CIT after each 
simulation run. As shown in Figure 10, participants strongly preferred the Integrated over the 
Standalone CIT (Standalone M- 1.7, SD = .89, Integrated M= 4.4, SD = .67; t(ll) = 6.42 , p = .001). 
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Figure 10. Controller CIT preference. 
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Additional questions were posed immediately after the completion of each run, following a video 
playback of the session when each CIT button was used. Each participant was asked “What did you 
use the [button name] for in this specific instance?” For each specific event, they were also asked, 
“Would you prefer the on-screen or off-screen CIT?” as well as, “Is there any additional functionality 
or information the CIT could have provided to you?” 

Users generally preferred the Integrated CIT when responding to preference questions, and in these 
cases cited situational awareness as the reason. The Integrated CIT permitted the user to keep their 
focus on the radar display. Observations showed that the Integrated CIT also seemed to allow 
information to be accessed faster than in the Standalone CIT in terms of visual scanning and trackball 
inputs. There were a few instances where the participants preferred the Standalone CIT when they 
wanted information such as SUAs and ground aids that could potentially clutter the radar display. 

4.3. CIT Use 

The participants’ ratings indicated a strong liking for the Integrated CIT. Expanding beyond 
preference, we examined the CIT usage to find out if differences occurred between the Standalone and 
Integrated CITs. 

The usage data showed that the ground aids, DEP, and SEA keys were used rarely or not at all. Both 
CITs displayed the boundaries of active SUAs and provided altitude information in the SUA’s label in 
red. Participants typically did not turn off SUAs and commented that the Integrated CIT made it easier 
to use SUA information. 

The remaining keys (frequency, sector, and some route keys (LAS, OAK, SFO, and SJC)) were used 
about the same between the Standalone and Integrated CITs. Figure 11 illustrates the average number 
of times that keys were pressed to toggle the various CIT components on and off for the two 
conditions. 

Statistical testing on the “key on” data from Figure 1 1 between the SUA, ground aids, frequency, 
sector, and route keys showed an overall main effect, (F(4, 50) = 124.25,/? = .0). Post hoc testing 
demonstrated that frequency was significantly different from SUA, ground aids, sector, and route, and 
sector was significantly different from SUA, ground aids, and frequency, but not route. 

There were no significant differences between the on and off usage for most of the keys. There 
appeared to be a large difference between the Standalone and Integrated conditions for frequency off 
and sector off. However, only the sector off results achieved statistical significance with Standalone 
CIT ( M= 1.4) compared to the Integrated CIT (M= 6.8), (t( 22) = 2.72 , p = .02). These controller 
actions suggest that they wanted to remove some information on the Integrated CIT in order to 
minimize display clutter. 
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Frequency key selection increased rapidly at the beginning of the scenarios and reached a high usage 
level by 10 minutes into the scenarios. (See Figure 12.) This use level was sustained throughout the 
test period. Frequency key activity appeared to be driven by operational necessity. 

Controllers required this information to successfully complete communication transfers for non- 
DataComm equipped aircraft prior to a hand-off to an adjacent sector. These frequency changes were 
required about 40 times in a scenario, creating a high demand for the data the key provided. 

Controllers indicated that the large amount of frequency data made the information difficult to retain in 
memory, which created the need to repeatedly access the key. 

The sector data were accessed less often than the frequencies. The sector key was used to support the 
frequency key in the majority of selections. The participants primarily chose the sector key to verify 
the vertical stratification of adjacent sectors, such as Salt Lake Center sectors 45 and 47, whose vertical 
boundaries were at a different altitude than sector 30/33’s vertical boundary. This key was used 
throughout the scenarios. (See Figure 13.) Controllers suggested that the sector and frequency keys be 
combined. (See Table 3.) 
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Figure 12. CIT entries: Frequency key initiated. 



Figure 13. CIT entries: Sector key initiated. 

While frequency keys were accessed throughout the simulation runs, the route keys were used heavily 
during the first five minutes of the scenarios and then at a much lower rate for the remainder of the 
time. (See Figure 14.) This suggests that the route information could be memorized within each 
scenario while the frequency information could not. Although the route information was complex in 
general, it seemed that the participants mainly needed to access specific data on altitude crossings on 
four of the six routes available on the CIT, which were easy to memorize. 
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Figure 14. CIT entries: Route key initiated. 

Figure 15 shows the percentage of time that the displayed data timed out at 15 seconds. Actively 
toggling off these data prior to the 15 seconds suggests that the participants wanted to minimize the 
display clutter caused by the data. In general, it appears that the keys in the Integrated display timed 
out less frequently (i.e., were actively toggled off more often) than the keys in the Standalone display. 
However, the differences were not statistically significant for the frequency and sector keys. The route 
keys were toggled off more often in the Integrated (M = 20%) compared to the Standalone CIT 
condition (M= 54%), (7(11) = 4.74 , p = .001). 

Figure 16 shows CIT key selections that were toggled on and off across different keys, averaged across 
the simulation runs. Although the participants preferred the Integrated CIT display, the results indicate 
that they accessed the information on these keys as often on the Standalone version. However, they 
actively toggled off the CIT information (rather than allowing it to time out after 15 seconds) more 
often with the Standalone CIT (M= 12.7) compared to the Integrated CIT (M= 26.8), (/(l 1 ) = 3.79 , p 
= .003), presumably to minimize radar display clutter. 

Table 3 represents the participants’ feedback on how displays and ease of use could be improved. 
Responses were requested by key type. Controllers recommended that the frequency and the route 
keys be combined because the information supplied by these keys was often required simultaneously. 
The majority of other responses suggested improvements in visibility and operationally related data. 
Omitted and “No change required” responses are not included in this table. 
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Figure 15. CIT time-outs. 



Figure 16. CIT key activity 
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Table 3. Number of participants per improvement recommendation. 


Item 

Frequency 

SUA 

Sector 

Route 

Farger font 

3 

1 

2 


Configurable font 

1 




Combine frequency and sector keys 

6 


4 


Make display persistent 

4 


1 

2 

Enhance UHF or VHF frequencies 

2 




Display use times 


2 



Add a list of future events 


1 



Add an SUA conflict probe 


1 



Add brightness control 


2 

1 


Allow data repositioning 



1 


Use different color 



1 


Display off-screen 




1 

Display miles-in-trail 




3 

Persistent altitude display 




1 

Separate miles-in-trail button 




1 

Drop route lines 




1 


4.4. Operational Issues 

A total of eight safety-related events occurred during the three days of testing. (See Table 4.) These 
events consisted of five operational errors, two operational deviations, and a pilot deviation. Each of 
these incidents was reviewed individually using Camtasia playback and voice recordings. 

Table 4. Safety-related events. 


Type of Error 

Standalone 

Integrated 

Operational Errors/Deviations 

4 

3 

Pilot Deviations 
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Conflict alert and conflict probe were not factors in the errors and deviations associated with the 
Standalone CIT. Controller scan, however, did seem to play a role. On two of the four Standalone 
errors/deviations, the controller was focused on completing other lower priority tasks or on another 
conflict. In each of these errors, the controller’s scan was diverted from the radar display to the 
Standalone CIT for information as the events were developing. In addition, the operational deviations 
in the Standalone CIT condition occurred due to aircraft penetrating the SUAs that were displayed on 
the Standalone display instead of being co-located with the aircraft on the radar display. 

Operational errors during scenarios where the Integrated CIT was in use were primarily attributed to 
controller difficulties using the automation tools. (See Figure 17.) The conflict probe and conflict 
alert functions failed to identify potential conflictions to the controller on three occasions. The 
conflict-detection functions were identical between the two CIT conditions, and yet the errors were 
attributed to different causes in the Standalone condition. Finally, one of the operational errors in the 
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Integrated CIT condition was related to the diversion of the controller’s scan to other areas of the 
display to address other conflicting traffic. 



Figure 17. Contributing factors to loss of separation events. (FDB = Full datablock) 

5. Discussion 

5.1. Workload 

This study was designed to gauge the participants’ ability to control traffic approaching the MAP value 
for ZOA sector 30/33 using features designed to reduce the need for training. DataComm and CD&R 
were also provided to help manage the task load for the controllers. Nevertheless, traffic workload and 
complexity level, coupled with the participants’ unfamiliarity with DataComm and CD&R, resulted in 
high overall workload assessments. 

At the conclusion of day 1, participant feedback indicated that additional training on these tools was 
required and an hour of instruction in DataComm and CD&R in MACS was added for the following 
three days of testing. Nonetheless, the participants’ workload assessments for the remaining days 
continued to be high. The high sector workload may have limited their ability to fully use the CIT. 

This suggests that, in research settings, proper training on advanced tools is required to ensure that 
controllers have the proficiency necessary to use them effectively. 

5.2. Controller Preferences 

The study shows that, based on subjective feedback in a controlled environment, users strongly 
preferred the Integrated CIT that placed the sector data on the radar screen. Use of the Standalone CIT 
seems to have resulted in a loss of visual attention to the main radar display. Participants tended to use 
keys that accessed specific information needed for managing traffic flow and communications 
transfers. Keys that gave general situational awareness of the sectors were used less often. 
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Participants normally viewed route data early in the scenario to ascertain the correct transition altitudes 
and did not need to access them further. Controllers were able to learn these altitudes quickly and 
apply the information correctly throughout the problems. Frequency information, however, needed to 
be accessed more often. The sector key was primarily selected to verify sector stratification. 

5.3. Factors that Impacted CIT Location 

Although the participants preferred having the CIT on the radar display, there is a limit on the amount 
of information that can be included before visual clutter becomes an issue. Good display design 
should be able to minimize this problem. 

In this study, the results suggested that clutter was a problem. The CIT was actively turned off more 
often when using the Integrated CIT. The results were most prominent for the route keys, indicating 
that they created the most clutter. When using the Integrated CIT, route data were viewed within the 
sector boundaries and had an impact on the controller’s traffic display. 

The factors that may play a role in determining the costs and benefits of displaying CIT data are: 

1 . Does the CIT create display clutter such that it interferes with air traffic control tasks? 

2. How often is the information needed? 

3. Does the information need to be viewed in the context of the traffic (e.g., locations of the SUAs 
in relation to the aircraft)? 

4. Should the CIT items always be on, or disappear after a period of time? 

Most of the frequency and sector data were located outside the sector boundaries, with the exception of 
sector 30/33 and 46 data. (Sector 46 was below sector 30/33.) These data were relatively unobtrusive 
and located away from the area being scanned by the controller, suggesting that they did not 
significantly interfere with the controllers’ tasks. Given that the frequency information was used most 
often, it seems to be a good candidate for inclusion on the radar display. 

Finally, controller situational awareness of active SUAs may have been impacted using the Standalone 
CIT. Although SUA data were displayed on both CITs during each run, the Standalone CIT was away 
from the controller’s scan. No memory aids existed in the controller’s field of vision to alert the 
controller to conflicting airspace. 

6. Conclusions 

A human-in-the-loop evaluation was completed to study Standalone and Integrated CITs. Participants 
were able to access information essential to successful air traffic operations in a busy, high altitude 
sector. The controllers strongly preferred the Integrated version of the CIT. The primary reason for 
this seemed to be that it allowed them to remain focused on the traffic situation, whereas the 
Standalone CIT required them to move their attention away from the radar presentation for short 
periods. 

The only detectable operational difference was that the location of the Standalone CIT might have 
contributed to some losses of separation. Participants accessed each CIT about equally, and the 
location of the data on-screen or off-screen did not inhibit their use. Although the information selected 
was similar for the two conditions, participants actively turned off the data on the Integrated CIT, 
presumably to reduce the clutter on the radarscope. 
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A caveat to the above results is that the high workload and complexity presented to the participants 
may have negatively impacted full use of the CITs. Information accessed during heavier traffic 
periods was limited to that deemed critical by the participant. Nevertheless, it was helpful to leam 
about controller use of the CIT under these conditions. Follow-up studies that explore CIT 
applications in a range of traffic situations would provide a fuller picture on the proper design of 
displays of sector information. 

The data chosen for these versions of the CIT were based on the results of previous human-in-the-loop 
studies and from controller subject matter expert feedback. Further work in this area is needed to 
identify what additional CIT data should be made available to controllers. It would be ideal to build a 
display architecture that is scalable, meaning that as data needs arise in the NextGen vision, new items 
can be added and presented to the controller. 
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